{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 统计分析\n",
    "\n",
    "通过指定统计分析字段，得到每个特征的p_value，所有的p_value计算都是基于Ttest计算。支持指定不同的分组`group`，例如train、val、test等分组统计。\n",
    "\n",
    "对于两大类不同的特征\n",
    "\n",
    "1. 离散特征，统计数量以及占比。\n",
    "2. 连续特征，统计均值、方差。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from onekey_algo import OnekeyDS as okds\n",
    "\n",
    "test_data = pd.read_csv(okds.survival)\n",
    "# 指定训练集、测试集，真实情况可以自己定义好\n",
    "val_ = int(test_data.shape[0] * 0.2)\n",
    "test_data['group'] = ['train'] * (test_data.shape[0] - val_) + ['val'] * val_\n",
    "test_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 输出格式\n",
    "支持两种格式数据，分别对应`pretty`参数的`True`和`False`, 当为`True`时，输出的是表格模式，反之则为dict数据。\n",
    "\n",
    "```python\n",
    "def clinic_stats(data: DataFrame, stats_columns: Union[str, List[str]], label_column='label',\n",
    "                 group_column: str = None, continuous_columns: Union[str, List[str]] = None,\n",
    "                 pretty: bool = True) -> Union[dict, DataFrame]:\n",
    "    \"\"\"\n",
    "\n",
    "    Args:\n",
    "        data: 数据\n",
    "        stats_columns: 需要统计的列名\n",
    "        label_column: 二分类的标签列，默认`label`\n",
    "        group_column: 分组统计依据，例如区分训练组、测试组、验证组。\n",
    "        continuous_columns: 那些列是连续变量，连续变量统计均值方差。\n",
    "        pretty: bool, 是否对结果进行格式美化。\n",
    "\n",
    "    Returns:\n",
    "        stats DataFrame or json\n",
    "\n",
    "    \"\"\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from onekey_algo.custom.components.stats import clinic_stats, clinic_stats_pluralism\n",
    "stats = clinic_stats(test_data, \n",
    "                     stats_columns=['result', 'gender', 'age', 'Tstage', 'BMI', 'drink', 'duration'],\n",
    "                     label_column='result', \n",
    "                     group_column='group', \n",
    "                     continuous_columns=['age', 'BMI', 'duration'], \n",
    "                     pretty=True)\n",
    "stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "sel_idx = [True if (isinstance(pv[0], str) and pv[0] != '') or (isinstance(pv[0], float) and pv[0] < 0.05) else False \n",
    "           for pv in np.array(stats['pvalue'])]\n",
    "test_data[stats[sel_idx]['feature_name']].to_csv('clinic_sel.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 输出dict类型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "stats = clinic_stats(test_data,\n",
    "                     stats_columns=['gender', 'age', 'Tstage', 'smoke', 'BMI', 'drink', 'degree', 'chemotherapy'],\n",
    "                     label_column='result', \n",
    "                     group_column='group', \n",
    "                     continuous_columns=['age', 'BMI'], \n",
    "                     pretty=False)\n",
    "\n",
    "print(json.dumps(stats, ensure_ascii=False, indent=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
